Introduction to biostatistics: Part 3. Statistical approaches to uncertainty: P values and confidence intervals unpacked.

(the difference in risk) is estimated to be 19.6% with a 95% CI of 5.7% to 33.6%. The P value of 0.006 means that an ARR of 19.6% or more would occur only in 6 in 1000 trials if streptomycin was equally as effective as bed rest. Since the P value is less than 0.05, the results are statistically significant (i.e., it is unlikely that streptomycin is ineffective in preventing death). The 95% CI suggests that the likely true benefit of streptomycin could be as small as 5.7% or as large as 33.6%, but is very unlikely to be 0% or less. Our best estimate for the ARR is 19.6% and hence the NNT is 6 (95% CI 3 to 18). This means that we might have to treat as many as 18 people with streptomycin or as few as 3 to prevent 1 additional person dying of tuberculosis. Figure 1. Clinical significance and statistical significance. Conventionally, a P value < 0.05 is taken to indicate statistical significance. This 5% level is, however, an arbitrary minimum and P values should be much smaller, as in the above study (P = 0.006), before they can be considered to provide strong evidence against the null hypothesis. Hence, reporting the exact P value (e.g., P = 0.027) is more helpful than simply stating that the result is significant at the 5% level (or 1% level, as above). If an effect is statistically significant, does this mean it is clinically significant? A statistically significant difference is not necessarily one that is clinically significant. In the above example, the statistically significant effect (P = 0.006) is also clinically significant as even a modest improvement in survival is important. For many effects, however, the benefit needs to be somewhat greater than zero for it to be of clinical significance (i.e., sufficiently beneficial to be worth the effort of treatment). In Figure 1, while both studies (a) and (c) show a statistically significant result, with the CIs not overlapping the " no difference " value, only (a) has a result that is consistent (in terms of the CI) with at least a minimum clinically important difference. Studies (b) and (d) are not statistically significant, as their CIs overlap the values of no difference. Minimum clinically important difference No difference (a) (b) I n t r o d u c t i o n What is statistical uncertainty? …